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Abstract 

Kullback-Leibler relative-entropy has unique properties in cases involving distribu- 
tions resulting from relative-entropy minimization. Tsallis relative-entropy is a one 
parameter generalization of Kullback-Leibler relative-entropy in the nonextensive 
thermostatistics. In this paper, we present the properties of Tsallis relative-entropy 
minimization and present some differences with the classical case. In the represen- 
tation of such a minimum relative-entropy distribution, we highlight the use of the 
q-product, an operator that has been recently introduced to derive the mathematical 
structure behind the Tsallis statistics. One of our main results is generalization of 
triangle equality of relative-entropy minimization to the nonextensive case. 
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1 Introduction 



Maximum and minimum entropy methods, known as ME methods, originally 
coming from physics, have been promoted to a general principle of infer- 
ence primarily by the works of Jaynes [1] and later by Kullback [2]. Jaynes 
maximum entropy principle involves maximizing Shannon entropy [3,4] while 
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KuUback minimum entropy principle involves minimizing KuUback-Leibler 
relative-entropy [2]. Logarithmic form of information measure is common for 
all these entropies. 

On the other hand, however, Tsallis in [5] proposed a non-logarithmic form of 
entropy (termed as noncxtcnsivc entropy or Tsallis entropy) which is consid- 
ered as a useful measure in describing thermostatistical properties of a certain 
class of physical systems that entail long-range interactions, long-term mem- 
ories and multi-fractal structures. The thermostatistical formalism based on 
Tsallis entropy is termed as generalized or nonextensive thermostatistics, since 
Tsallis entropy is a one-parameter generalization of Shannon entropy and does 
not satisfy additive property involving independent probability distributions 
but satisfies the so-called pseudo additivityoi nonextensive additivity (see (9)). 
These generalized statistics have been applied not only to physical systems but 
also to various problems in optimization (generalized simulated annealing [6]), 
statistical inference [7], machine learning [8] etc. Recently Shannon-Khinchin 
axioms have been generalized to nonextensive systems [9]. 

On similar lines, generalization of relative-entropy, called Tsallis relative-entropy, 
has been proposed in [7]. In this paper, we study Tsallis relative-entropy 
minimization and its differences with the classical case. We generalize the 
triangle equality (see (6)) of Kullback-Leibler relative-entropy minimization, 
which qualifies relative-entropy minimization as an optimal inference proce- 
dure with respect to relative-entropy as an information measure, to Tsallis 
relative-entropy case. 

We present the necessary background in § 2, where we discuss properties 
of relative-entropy minimization in the classical case, and we give the basic 
definitions related to Tsallis entropy. In § 3 we present the relative-entropy 
minimization in non-extensive framework and discuss its differences with the 
classical case. Finally, triangle equality for Tsallis relative-entropy minimiza- 
tion is derived in § 4. A brief discussion of Tsallis relative-entropy minimization 
in the case of "normalized g-expected" values is presented in § 5. 



2 Background 

2.1 Relative- entropy minimization: In classical case 

Minimizing the Kullback-Leibler relative-entropy (or I-divergence or cross- 
entropy) with respect to a set of moment constraints finds its importance in the 
celebrated Kullback's minimum relative- entropy principle [2]. This principle 
is a general method of inference about an unknown probability distribution 
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when there exists a prior estimate of the distribution and new information in 
the form of constraints on expected values [10]. Formally, we can state the 
minimum relative-entropy principle as: given a prior distribution r, of all the 
probability distributions that satisfy the given moment constraints, one should 
choose the posterior p with the least relative-entropy 

h(p\\r)^ [dxp{x)\n^ , (1) 



provided that the integral above exists . The prior distribution r can be a 
reference distribution (uniform, Gaussian, Lorentzian or Boltzmann etc.) or a 
prior estimate of p. 

The principle of Jaynes maximum entropy is a special case of minimization 
of relative-entropy under appropriate conditions [13]. In particular, minimiz- 
ing relative-entropy is equivalent to maximizing Shannon entropy when the 
prior is a uniform distribution. Relative-entropy minimization has been ap- 
plied primarily to statistics [2], and also to statistical mechanics [14], pattern 
recognition [15], spectral analysis [16], speech coding [17]. For a list of refer- 
ences on applications of relative-entropy minimization see [13,12]. 

Properties of relative-entropy minimization have been studied and presented 
extensively in [10]. Here we briefly mention a few. Given a prior distribution 
r with a finite set of moment constraints of the form 

j dxum{x)p{x) = {Um) , m = 1, . . . M , (2) 



along with the normalizing constraint / p{x)dx = 1 (from now on we assume 
that any set of constraints on probability distributions implicitly includes this 
constraint) , the minimum relative-entropy distribution is of the form 



r(x)e Sm=i '^'""'"(^) 



p(x) = , (3) 

^1 



where 

Zi = yda;r(x)e-S-=i^'""'"(^) (4) 

^ In measure theoretic terms, the integral exists if the measure induced by p is abso- 
lutely continuous with respect to that induced by r, otherwise /i(]3||r) = oo [11]. In 
this work wc do not aim at mathematical rigor of the measure theoretic information 
theory. In particular, we assume that all quantities of interest exist for all distri- 
butions considered. Note that the measure theoretic definitions of these quantities 
relies strongly on the Lebesgue- Radon- Nikodym Theorem [12]. 
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is the partition function, Z^^, m = 1, . . . , M are the corresponding Lagrange 
multiphers, u„, . m = 1, . . . M are some functions of the underlying random 
variable whose expectation values (um), m = 1, . . . M are (assumedly) a priori 
known. When the prior is a uniform distribution with a compact support of 
W {W possible configurations in discrete case), the minimum relative-entropy 
distribution turns out to be 

P(^) = 7 ^^ir- — > (5) 



which is in fact a maximum entropy distribution (Boltzmann distribution) of 
Shannon entropy with respect to the constraints (2). 

Many properties of relative-entropy minimization just reflect well-known prop- 
erties of relative-entropy but there are surprising differences as well [10]. For 
example, relative-entropy does not generally satisfy a triangle relation involv- 
ing three arbitrary probability distributions. But in certain important cases in- 
volving distributions that result from relative-entropy minimization, relative- 
entropy does satisfy triangle equality. 

The statement of triangle equality can be formulated as follows. Let r be 
the prior distribution, p be the probability distribution that minimizes the 
relative-entropy subject to set of constraints (2) and I be any other distribution 
satisfying the same constraints, then we have the triangle equality [10]: 

hm - hm + h{p\\r) . (6) 



This triangle equality is important for application in which relative-entropy 
minimization is used for purposes of pattern classification and cluster analy- 
sis [15]. 



2.2 Nonextensive framework 



Tsallis entropy, which was introduced by Tsallis [5] is given by 

S,{p) = - / dxp{xf-^^^^^^ , (7) 



where g e M is called non-extensive index (g is positive in order to ensure the 
concavity of Sq). Tsallis entropy is a one-parameter generalization of Shannon 
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entropy in the sense that 

lim5,(p) = -/rfxpWl„pW = ft(p) 



(8) 



The cntropic index q characterizes the degree of nonextensivity reflected in 
the pseudo-additivity property 



S,{A + B)^ S,{A) + S,{B) + (1 - q)S,{A)S,{B) , 



(9) 



where A and B are two independent systems in the sense that the probabihty 
distribution oi A + B factorizes into those of A and B. 

Maximizing the TsaUis entropy Sg with respect to the constraints 

j dxum{x)p{xy ^ {Um)g , m = 1, . . . M , (10) 



the generahzed equihbrium probabihty distribution is found to be [18], 

M 

1 - (1 - PmUm{x) 



p{x) 



m=l 



1-9 



(11) 



where 



dx 



M 



1- {l-q)J2 PmUm{x) 



m=l 



1 

1-9 



[12) 



is the partition function, and Z^^, m = 1, . . . M are the corresponding Lagrange 
multiphers^ and {um)q is a known q-expectation of Um [18]. This distribution 
is called generalized maximum entropy distribution or simply TsaUis distribu- 
tion [19]. The limit g ^ 1 in (11), recovers the maximum entropy distribution 
in the classical case. 



^ To avoid proliferation of symbols we use same notation for the minimum or max- 
imum entropy distributions and Lagrange multipliers in the various cases; the cor- 
respondence should be clear from the context. 
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Now, the definition of Kullback-Leibler relative-entropy /i (1) and the gener- 
ahzed entropic form Sg (7) naturally lead to the generalization [7] 



Iq{p\\f) = j dxp{x)- 



r{x) 



9-1 



- 1 



q-1 



(13) 



which is called as TsaUis relative-entropy. The hmit g — > 1 recovers the 
relative-entropy in the classical case. Also one can verify that (see [7] ) 



lq{p\\r)>0 if g > 
= ifg = 
< if g < 



(14) 



For q ^ 0, the equalities hold if and only if p = r almost everywhere. Fur- 
ther, for q > 0, Iq{p\\r) is a convex function of p and r, and for g < it is 
concave [20] . Like Tsallis entropy, TsaUis relative-entropy satisfies the pseudo- 
additivity of the form [21] 



Ig{A^ + A2\\B^ + B,) = +^,(^211^2) 

+{q-l)Ig{A,\\B,)Ig{A2\\B2) , (15) 



where Ai, A2 and Bi, B2 are the independent pairs. The limit g — > 1 in (15) 
retrieves 



hiA, + A2\\B, + B2) = IM.WB,) + hiA2\\B2) 



(16) 



the additivity property of Kullback-Leibler relative-entropy. 



Further properties of Tsallis relative-entropy have been discussed in [7,20,21]. 
Characterization of Tsallis relative-entropy, by generalizing Hobson's unique- 
ness theorem [22] of relative-entropy, is presented in [23]. 
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3 Tsallis relative-entropy minimization 



3.1 Generalized minimum relative- entropy distribution 



To minimize Tsallis relative-entropy with respect to the set of constraints (10) 
the concomitant variational principle can be written as 

5 |/,(/>||r) + A (y dxp{x) -l^ + J2f^m[J dxp{xfum{x) - {um)}j 1=0, 



where A and (3m, m = 1, . . . M arc Lagrange multipliers. This gives us minimum 
Tsallis relative-entropy distribution as [20] 



p{x) 



r{x) 



l-q 



M 



UmAX 



m=l 



1 

1-9 



(17) 



where values of Pm, m — 1 . . . M are determined by the constraints (10) and 
Zq, the partition function, is given by 



dx 



M 



r{x) * - (1 - g) E PmUm{x) 



m=l 



1 

1-9 



(18) 



A note on the constraint (10). This constraint had been used for some time [24], 
but because of problems in justifying it on physical grounds the constraint 



J dxp{xyUm{x) 



dxp{x) 



(19) 



has been introduced in [18]. {{um))g is called normalized q-expectation value of 
Um- We discuss Tsallis relative-entropy minimization and its properties with 
respect to the constraint (19), briefly, in § 5. 



3.2 q-product representation of generalized minimum relative- entropy distri- 
bution 



The mathematical basis for Tsallis statistics comes from the q-deformed ex- 
pressions for the logarithm {q-logarithm) and the exponential function {q- 
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exponential) which were first proposed in [25], in the context of nonextensive 
thermostatistics. The g-logarithm is defined as 



- 1 

luqx^ 1 _ g (x>0,geM), (20) 



and the g-exponential is defined as 



[1 + (1 - q)x]^ iil + (l-q)x>0 
otherwise. 



These two functions are related by 

ej^«^ = x . (22) 



Properties of these q-deformed functions are studied in [26]. In this framework 
a new multiplication operation, called g-product 



x®qy=< 



ix^-'i + y^-'i - 1) if z, 2/ > and + - 1 > 
otherwise. 



is first introduced in [27] and explicitly defined in [28] for satisfying the fol- 
lowing equations: 



\nq{x®qy)^\nqX^\nqy , (24) 
e^®.e^ = ef^. (25) 

The g-product recovers the usual product in the limit q ^ 1 i.e., limg^i(x ®q 
y) = xy. The fundamental properties of the g-product ®g are almost the same 
as the usual product, and in general 

a{x y) ^ ax®qy (a, x, e R) . 



Further properties of the g-product can be found in [27,28]. Also, g-product 
has been used in various apphcations of TsaUis statistics [29] . 

Previously, generalized entropies and maximum entropy distributions have 
been represented in terms of g-logarithm and g-exponential. We list some of 
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them which we are going to use later in this paper. TsaUis entropy (7) can be 
represented as 



Sq{p) = - j dxp{xfhlqp{x) , 



(26) 



and TsaUis relative-entropy (13) as 



/t{x\ 
dxp{x) lug-^ 



(27) 



One can represent TsaUis distribution (11) in terms of q-exponential as 



p{x) 



(28) 



since p{x) — whenever 1 — {1 — q) J2m=i PmUmix) 
cut-off condition [5] assumed imphcitly. 



< which is TsaUis 



Note that generahzed rclativc-cntropy distribution (17), is not of the form 
of (3) even if we replace the exponential with the q-exponcntial. But one can 
verify the non-trivial fact that (17) can be expressed in a similar form as in 
the classical case by invoking q-product as, 



p{x) 



r{x) <Siq Bq 



(29) 



where 



dxr{x) ®q Cc 



One can see from the later parts of this paper, this representation is useful in 
deriving properties of TsaUis relative-entropy minimization. 



It is important to note that the distribution in (17) could be a (local/global) 
minimum only if g > and TsaUis cut-off condition is extended to the relative- 
entropy case i.e., p{x) — whenever r{xY~'^ — (1 — g) Ylm=\ PmUmix) < 0. 
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3.3 Properties 



As we mentioned earlier, in the classical case i.e., when g = 1, relative-entropy 
minimization with uniform distribution as a prior is equivalent to entropy 
maximization. But, in the case of nonextensive framework, this is not true. 
Let r be the uniform distribution with compact support W over C M. Then, 
by (17) one can verify that probability distribution which minimizes Tsallis 
relative-entropy is 



p{x) 



M 



m=l 



1-9 



/ 



dx 



M 



- (1 - Z] PmUm{x) 



m=l 



1 ) 

1-9 



which can be written as (by (39) and (42)) 



p{x) 



I 

E 



dx e. 



(30) 



or (by the definition of g-exponential (21)) 



p{x) = 



I 



dx e, 



(31) 



By comparing (30) or (31) with Tsallis maximum entropy distribution (28) 
one can conclude (formally one can verify this by thermodynamical equations 
of Tsallis entropy [5]) that minimizing relative-entropy is not equivalent^ to 
maximizing entropy when the prior is uniform distribution. The key observa- 
tion here is W appeares in (31) unlike (28). 



^ For fixed g-expected values {um)q, the two distributions, (31) and (28) are equal, 
but the values of corresponding Lagrange multipliers are different when q ^ 1 (while 
in the classical case they remain same). Further, (31) offers the relation between the 



Lagrange parameters in these two cases. Let (3, 



(5) 



m 



1, . . . M corresponds to the 



Lagrange parameters corresponds to the generalized maximum entropy distribution 



while /3, 



m 



1, . . . M corresponds to generalized minimum relative-entropy dis- 



tribution with uniform prior. Then, we have relation (3^^ = W^-''p^\m = 1, . . . M. 
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Also minimum Tsallis relative-entropy satisfies [20]: 

M 

/,(p||r) = - In, - ^ Pm{um), (32) 

m=l 

to prove which, one can escape lengthy algebraic manipulations by using q- 
deformed representations of various formulae. 

The thermodynamic equations for the minimum Tsallis relative-entropy are 

{u^)q , m = 1,...M , (33) 
Pm , m = l,...M , (34) 

which generalize thermodynamic equations in the classical case. One should 
note that these thermodynamic equations were proved to hold true for, essen- 
tially, any entropic measure [30,31]. 



4 Nonextensive triangle equality 

Before we derive equivalent of triangle equality in the nonextensive thermo- 
statistics we shall discuss the significance of triangle equahty of KuUback- 
Leibler relative-entropy minimization. Significance of triangle equality comes 
in the following scenario. Let r be the prior estimate of the unknown prob- 
ability distribution / about which information in the form of expected value 
constraints 

j dxum{x)l{x) — {um) , m — 1, . . . M (35) 



d 



lUq Zq = - 

dL 



is available for fixed functions u^, rn — 1, . . . M. The problem is to choose 
a posterior estimate p that is in some sense the best estimate of / given the 
available information i.e., prior and the information in the form of expected 
values. The principle of minimum entropy provides a general solution to this 
inference problem and provides us the estimate (3). Further, from triangle 
equality (6), the minimum relative-entropy posterior estimate of / is not only 
logically consistent, but also closer to /, in the relative-entropy sense, than is 
the prior r. Moreover, the difference /i(/||r) — Ii{l\\p) is exactly the relative- 
entropy /i(p||r) between the posterior and the prior. Hence Ii{p\\r) can be 
interpreted as the amount of information provided by the constraints that is 
not inherent in r. 
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Additional justification to use minimum relative-entropy estimate of p with 
respect to constraints (2) is provided by the following expected value matching 
property [10]. For fixed functions Um, m = 1, . . . M, let the actual unknown 
distribution / satisfy 

J dxUm{x)l{x) — (Wm) , m — 1, . . . M . (36) 



Now, as (um), m — 1, . . . M vary, Iq{l\\p) has the minimum value when 

{Um) = {Wm) , m=l,...M. (37) 



For the proof of expected value matching property see [10]. This property 
states that for a distribution p of the form (3), Ii{l\\p) is the smallest when 
the expected values of p match those of /. In particular, p is not only the 
distribution that minimizes /i(j9||r) but also is the distribution of the form (3) 
that minimizes Ii{l\\p). This property is a generalization of a property of or- 
thogonal polynomials [32] which in the case of speech analysis [17] is called 
the "correlation matching property" . 

From the above discussion, it is clear that to derive a triangle equality of 
Tsallis relative-entropy minimization, one should first deduce the equivalent 
of expectation matching property in the nonextensive case. Let / be the actual 
unknown distribution which satisfies 

j dxUm{x)l{xY = {Wm)q ■, Ul = 1, . . . M , (38) 



r be the prior estimate of / and p be the posterior which satisfies constraints 
(10). That is, we would like to find the values of {um)g for which Iq{l,p) is 
minimum. We write the following useful relations before we proceed to the 
derivation. 



We can write generalized minimum relative-entropy distribution (29) as 

p{x) = = , (39) 



by (22) and (25). Further by using 

lng{xy) =lngX + In^ y + {1 - q) In^ x In^ y (40) 
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we get the relation 

M 

\nqp{x) + lllq + (1 - g) Ingp(x) lllq = " X] PmUmix) + In^ r{x) .(41) 

m=l 



By the property of g-logarithm [33] 

ln,(^^^=y''-\ln,x-ln,y) . (42) 

and by (26), (27) one can verify that 

iMr) = - / dxp{xy]n,r{x) - Sg{p) . (43) 

To proceed with the derivation, consider 



^qi^Wp) = - j dxl{x)\ng 
By (42) we have 



l{x) ■ 



h{A\p) = - j dxl{xy \hiqp{x) -lnq/(a;)] 

^ISW)- j dxl{xy[\nqp{x)-\ugr{x)] . 

From (41), we get 



(44) 



/,(/|b) =/,(/||r)+ jdxl{xf [Y.P^^rn{x)j+\n,Zqjdxl{xf 

+(1 - g) S j dx l{xy Xuq p{x) . (45) 

By using (38) and (43), 



M „ 

I,{1\\P) = Iq{A\r) + E f^m{Wm), + K / dxl{xf 
m=l ■' 



+ (l-g)ln,Z,[-/,(/|b)-5,(/)] , (46) 
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and by (7) we have 



M 



IMp) = iMr) + E Pm{wm), + H Z,-(l- q) H V.(^lb) ■ (47) 



771=1 



Since the muhiphers Pm, m = 1, . . . M are functions of the expected values 
{U"m)q, variations in the expected values are equivalent to variations in the 
multipliers. Hence to find the minimum of Iq{l,p), we solve 



which gives us 

.rn=l,...M. (48) 

In the limit g — >• 1 the above equation gives (tim)i = {wm)i which is the 
expectation matching property in the classical case. 

Now, to derive the triangle equality for Tsallis relative-entropy minimization, 
we substitute the expression for {wm)q, which is given by (48), in (47). And 
after some algebra one can arrive at 

iMr) = IMp) + iMr) + {q- l)lMp)m\r) ■ (49) 

The limit g ^ 1 in (49) gives the triangle equality in the classical case (6). 
The two important cases which arise out of (49) are, 

/,(/||r) < IMp) + lM\r) when < g < 1 , (50) 
lM\r) > IM\P) + IiiP\\r) when Kg. (51) 

We call (49) as nonextensive triangle equality, whose pseudo additivity is con- 
sistant with the pseudo additivity of Tsallis relative-entropy (compare (15) 
and (16)), and hence is a natural generalization of triangle equality in the 
classical case. 



5 In the case of 'normalized g-expectations' 



In this Section we discuss Tsallis relative-entropy minimization with respect to 
the constraints in the form of normalized g-expectations (19). For a complete 
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discussion on choice of constraints (10) and (19) for Tsallis entropy maximiza- 
tion see [18,34]. 

The variational principle for Tsallis relative-entropy minimization with respect 
to (19) can be written as 



S !^lM\r) + A (/ dxp{x) - l) + 



^ ^/ dxp{xYu{x) 



J dxp{xy 



where A and /S^, m = 1, . . . M are Lagrange multipliers. This gives generalized 
minimum relative-entropy distribution as 



p{x) 



t[x) "-(l-f/) f- — 



(52) 



where 



Zf, = dx 



rix 



1-9 



/ dxp{xy 



1 

1-9 



(53) 



q ^ 1 in (52) retrieves the minimum relative-entropy distribution in the clas- 
sical case. 



This can be expressed as 

" Em=l {^m 



p{x) 



(54) 



where 



J dxp{x) 



, m = 1,...M . 



(55) 



Minimum Tsallis relative-entropy in this case satisfies 



(56) 



while corresponding thermodynamical equations can be written as 

— ^1-0 

^ Note that unlike Tsallis entropy case [18], J dxp{xy ^ Zq 
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dp, 



d ^ 

\ngZg = -{{um))„ , m = l,...M , (57) 



d{{um)) 
where 



dl 

' ^-Pm , m = l,...M , (58) 



M 



lug Zg = In^ Zg- J2 Pm{{um))g ' (59) 



m=l 



Now using above relations one can prove that, in this case too, Tsallis relative- 
entropy satisfies non-extensive triangle equality with modified conditions from 
the case of ^-expectation values. We state it formally as follows. Let r be the 
prior estimate of the unknown distribution I which satisfies 

JdxUm{x)l{xy 

= ((«^m))„ , m= 1,...M , (60) 



jdxi{xy 



where {{wm))g, rn — 1, . . .M are known normalized g-expected values of /. 
Let p be the posterior which satisfies constraints (19). Then, similar to the 
calculations in § 4 one can prove that Tsallis relative entropy satisfies the 
nonextensive triangle equality (49), provided 

{Mg = {{wm))g m = l,...M , (61) 



but the minimum of Iq{l\\p) is not guaranteed. Note that this condition is same 
as expectation value matching property in the classical case (see (37). 

The detailed study of Tsalhs relative-entropy minimization in this case of nor- 
malized g-expected values and the computation of corresponding minimum 
relative-entropy distribution (where one has to address the self-referential na- 
ture of the probabilities p{x) in (52)) based on TsaUis et. al [18], Martinez et. 
al [34] formalisms for Tsallis entropy maximization is under study. 



6 Conclusions 



Tsallis relative-entropy minimization has been studied and some significant 
differences with the classical case are presented. Generalized relative-entropy 
minimization has been shown to satisfy an appropriate generalized version of 
triangle equality for the classical case. This is yet another remarkable and 
consistant generalization shown by Tsalls statistics. Considering the various 
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fields to which TsaUis generahzed statistics has been apphed, studies of apph- 
cations of TsaUis relative-entropy minimization for various inference problems 
are welcome. 
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